Display control system

ABSTRACT

A display control system for a multi-layer display system including at least one multi-layer display unit. A multi-layer display unit includes multiple stacked or overlapped display layers each adapted to render display elements thereon for viewing through the uppermost display layer. The level of difficulty for a user to view an overlapped image differs depending on the color, brightness or type of images (for example text-on-text) being displayed and also the layer in which the image is being displayed and whether the image is currently active. At least any overlapping portion of display elements from different display layers are adjusted (automatically or by a user) in order to improve the viewing experience for a viewer. Types of adjustment possible include moving the images within their display layer or to an alternative display layer or altering their visual properties (such as color, brightness, contrast, etc.).

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of the InternationalApplication Serial Number PCT/NZ04/000092, having an InternationalFiling Date of 17 May 2004, which claims priority to New Zealand PatentApplication No. 525956, filed on 16 May 2003, both of which are hereinincorporated by reference in their entireties. Further, this applicationis a continuation of and claims priority to and the benefit of U.S.patent application Ser. No. 13/443,764, entitled “DISPLAY CONTROLSYSTEM, filed on Apr. 10, 2012, which is incorporated herein byreference in its entirety; which claims priority to and the benefit ofU.S. application Ser. No. 10/557,157, entitled “DISPLAY CONTROL SYSTEM,”with filing date Mar. 5, 2007, which is herein incorporated by referencein its entirety; both of which claim priority to and the benefit of theabove referenced International Application Serial Number PCT/NZ04/000092and New Zealand Patent Application No. 525956.

TECHNICAL FIELD

This invention relates to a display control system and more particularlythough not solely to a method and apparatus for simplifying theoperation of and/or improving a user's viewing experience with amulti-layer display system (a display system consisting of onemulti-layer display (MLD) device, multiple interconnected MLD devices,or interconnected single layer display (SLD) devices and/or MLDdevices).

BACKGROUND ART

Multi-layered displays provide a significant improvement over existingsingle layer displays (SLD) or screens. MLD units may be used to nestdisplay content over specially displaced or stacked or sandwiched layersto provide an enhanced mechanism for information absorption and analysisby users. An example of an existing multi-layer display is discussed forexample in WO9942889A in which each layer may be, for example, aselectively transparent Liquid Crystal Display (LCD) layer having theability to display images of adjustable transparency.

In commonly available SLD and MLD units, a backlighting system isprovided behind the display layer or layers. The resultant imageprovided to a user of the display is the result of selective filtering,by the display layer or layers, of the light produced by the backlightbefore it reaches the user's eyes. In an SLD unit, display elements(such as GUI windows, images icons or even pixels) within the singledisplay layer may be overlaid at will and software utilised to controlthe appearance of the display elements. For example, in Microsoft®'sWindows® operating system, various windows may be overlapped and thecurrently “in focus” (or active, or highlighted or “foreground”) windowwill always be shown in its entirety whereas any overlapped or occludedportion of another window will effectively be cut off or hidden, behindthe in focus or foreground window. This provides a very basic illusionof depth to an SLD unit.

However, in an MLD unit, because images on all but the rearmost displaylayer are at least partially transparent, overlapping of displayelements can lead to viewing difficulties. The degree of the viewingdifficulty is dependent upon whether the respective display elements areassigned to different display layers, the colour or contrast of therespective display elements and also depends on which of the displayelements are currently in focus. For example, in a two layer MLD unit, adark image on the rear display layer (the layer closest to thebacklighting system or rear of the screen) which overlaps with a displayelement on the front display layer, will reduce the intensity of lightavailable to illuminate the front layer display element. Accordingly,even if the display element on the front layer is in focus, theappearance to a user of its overlapped portion will be altered (that is,it will be darker or dimmer or even a different colour than intended) incomparison to its non-occluded portion. In some circumstances, this canmake it virtually impossible for a user to correctly or comfortably viewthe foreground display element.

There are also known display technologies for display layers in whichthe layer itself emits light rather than utilising a backlight. Thistechnology is in its infancy and although the aforementioned problem ofa dark or opaque rear display element reducing the available light foran overlapped front display element is reduced or eliminated, there isstill a problem in that the appearance of overlapping portions ofdisplay elements in different layers can interfere, thereby reducing thelegibility or discernability of the information displayed (wherelegibility relates to the distinctness that makes perception or readingeasy).

An example of the aforementioned problem occurs when two displayelements such as GUI windows, both displaying text, are positioned overeach other (that is, they overlap) in separate display layers of amulti-layer display device. The result is that the text on both layersis very difficult (if not impossible) to read. “Text-On-Text” is thename which we use to describe the problem however it is not limited tosituations involving only text. Overlaying text on a graphic image (orvice versa) can have the same effect of rendering the text difficult toread. More specifically, the problem is at its worst:

-   -   when there is very little contrast (that is, difference in        colour, pattern, brightness etc) between the overlapped display        elements or images displayed on different display layers, and/or    -   when at least one display layer is displaying cluttered        information.

In some instances, to assist in the effective display of information tousers it is of advantage to swap particular components of displaycontent between the layers of the MLD units. This allows high priorityinformation or information of high importance to be presented at theforefront of an observer's area of observation. This facility alsoallows a degree of control and flexibility with respect to howparticular groups of display elements (for example GUI windows) may benested and collected together for the consideration of relatedinformation.

Conventionally however, MLD units have been operated using software forcontrolling interconnected SLD units. For example, two single layerdisplay devices may be connected to a single video output of a computerfor example, and the screens are combined so that the row of pixelsalong one edge (for example the right hand edge) of a first device areconsidered to be immediately adjacent to the row of pixels along oneedge (for example the left hand edge) of the second device. In this way,movement of a cursor or window past the right hand edge of the firstdevice causes the cursor or image to appear at the left hand edge of thesecond device as if the two display devices were a single display layerin the same plane.

In an MLD unit, the separate display layers are instead stacked inparallel planes but the same software for manipulating objects ordisplay elements is conventionally employed. Accordingly, in order for auser currently working in a first display layer to move a displayelement from a second display layer to the first display layer of an MLDunit, it has conventionally been necessary for the user to first movetheir mouse pointer to an edge of the first display layer and on to thesecond display layer, select the desired display element and either dragthat element back across the second display layer and on to the firstlayer or, once the display element has been selected, alter itsproperties in such a way that it is automatically shifted by appropriatesoftware to the first display layer.

An improved method, system, apparatus or computer software whichmonitored and automatically improved the legibility or discernability or“viewability” of display elements in an MLD system and/or assisted usersin assigning display elements to layers or in transferring displayelements between layers of an MLD system would be of advantage. It wouldbe preferable to have hardware or software code or a set of algorithmicinstructions programmed into a computer system or other types ofprogrammable logic devices, which when run, address the issues discussedabove.

All references, including any patents or patent applications cited inthis specification are hereby incorporated by reference. No admission ismade that any reference constitutes prior art. The discussion of thereferences states what their authors assert, and the applicants reservethe right to challenge the accuracy and pertinency of the citeddocuments. It will be clearly understood that, although a number ofprior art publications are referred to herein, this reference does notconstitute an admission that any of these documents form part of thecommon general knowledge in the art, in New Zealand or in any othercountry.

It is acknowledged that the term ‘comprise’ may, under varyingjurisdictions, be attributed with either an exclusive or an inclusivemeaning. For the purpose of this specification, and unless otherwisenoted, the term ‘comprise’ shall have an inclusive meaning—i.e. that itwill be taken to mean an inclusion of not only the listed components itdirectly references, but also other non-specified components orelements. This rationale will also be used when the term ‘comprised’ or‘comprising’ is used in relation to one or more steps in a method orprocess.

It is therefore an object of the present invention to provide a displaycontroller or display control method or display layer transfer methodwhich will go at least some way towards addressing the foregoingproblems or which will at least provide the public or industry with auseful choice.

Further aspects and advantages of the present invention will becomeapparent from the ensuing description which is given by way of exampleonly.

DISCLOSURE OF INVENTION

Accordingly, in a first aspect the invention consists in a displaycontrol method for a multi-layer display system including at least twooverlapping display layers, the display control method comprising thesteps of:

-   -   i) detecting that the position of a first display element to be        displayed on a first display layer overlaps or will potentially        overlap with the position of a second display element on a        different display layer,    -   ii) adjusting either the position of the first and/or second        display element and/or at least one visual property of at least        the overlapping or potentially overlapping portion of the first        and/or second display element, and    -   iii) displaying the first and second display elements on their        respective display layers in their adjusted position or        positions or with their adjusted visual property or properties        in order to improve the ability of a user of the multi-layer        display system to view the overlapping or potentially        overlapping portion of the first and/or second display element.

Preferably, the step of adjusting is carried out manually by a user.

Alternatively, the step of adjusting is carried out automatically.

Preferably, the step of automatically adjusting comprises:

-   -   determining a level of interference as experienced by a user        between the first and second display elements, and    -   moving the first and/or second display element within their        respective display layers, to a new position or positions which        reduces the determined level, and/or    -   changing at least one visual characteristic of the first and/or        second display element.

Preferably, the step of adjusting the position of a first and/or seconddisplay element comprises moving the first and/or second display elementwithin their respective display layers, so that there will be no overlapor reduced overlap between the first and second display elements.

Preferably, the method also includes the step of determining whether thefirst or second display element is currently active and the step ofadjusting is carried out dependent upon this determination.

Preferably, the first display layer is nearer to the front of themulti-layer display system than the second display layer and

-   -   if the first display element is active and the second display        element is not active, then at least one visual property of at        least the overlapping or potentially overlapping portion of the        second display element is adjusted, or    -   if the second display element is active and the first display        element is not active, then at least one visual property of at        least the overlapping or potentially overlapping portion of the        first display element is adjusted.

Preferably, the step of adjusting comprises adjusting at least one ofcolour, saturation, brightness or transparency of at least theoverlapping or potentially overlapping portion of the first and/orsecond display element or the contrast between overlapping orpotentially overlapping portions of the first end second displayelements.

Preferably, the step of adjusting comprises combining at least onevisual property of the overlapping or potentially overlapping portionsof the first and second display elements and the resultant combinedvisual property is applied to the overlapping portion of the activedisplay element while at least one visual property of the other displayelement is adjusted to minimise its effect on the image provided by theactive display element.

Alternatively, the step of adjusting comprises adjusting at least onevisual property of at least the overlapping portion of both the firstand second display elements.

Preferably, the method also comprises the step of allocating the firstand/or second display element to a selected display layer or layersbased upon the at least one determined visual property or upon anassociated pre-set display layer preference.

Preferably, the step of allocating is based upon stored preferencesassociated either with a user of the multi-layer display system or withparticular software application programs run on a computer system towhich the multi-layer display system is connected.

Preferably, when a particular display layer includes a plurality ofdisplay elements, the display elements are arranged within that displaylayer to thereby control the way in which overlapping portions ofdisplay elements in a single display layer are displayed.

Preferably, when a plurality of display elements are dispersed across atleast two display layers, the display elements are arranged over the atleast two display layers thereby controlling the way in whichoverlapping portions of display elements on different display layers aredisplayed.

Preferably, the colours of at least the overlapping portion of the firstand second display elements are adjusted on a pixel-by-pixel basis.

In a second aspect, the invention consists in a display controller for amulti-layer display system including at least two overlapping displaylayers, the display controller comprising:

-   -   i) means for detecting that the position of a first display        element to be displayed on a first display layer overlaps or        will potentially overlap with the position of a second display        element on a different display layer,    -   ii) means for adjusting either the position of the first and/or        second display element and/or at least one visual property of at        least the overlapping or potentially overlapping portion of the        first and/or second display element, and    -   iii) means for displaying the first and second display elements        on their respective display layers in their adjusted position or        positions or with their adjusted visual property or properties        in order to improve the ability of a user of the multi-layer        display system to view the overlapping or potentially        overlapping portion of the first and/or second display element.

Preferably, the means for adjusting are operated manually by a user.

Alternatively, the means for adjusting operate automatically.

Preferably, the means for automatically adjusting the position of afirst and/or second display element comprises:

-   -   means for determining a level of interference as experienced by        a user between the first and second display elements, and    -   means for moving the first and/or second display element within        their respective display layers, to a new position or positions        which reduces the determined level, and/or    -   means for changing at least one visual characteristic of the        first and/or second display element.

Preferably, the means for adjusting the position of a first and/orsecond display element moves the first and/or second display elementwithin their respective display layers, so that there will be no overlapor reduced overlap between the first and second display elements.

Preferably, the display controller also comprises means for determiningwhether the first or second display element is currently active andproviding this information to the means for adjusting.

Preferably, the first display layer is nearer to the front of themulti-layer display system than the second display layer and

-   -   if the first display element is active and the second display        element is not active, then at least one visual properly of at        least the overlapping or potentially overlapping portion of the        second display element is adjusted, or    -   if the second display element is active and the first display        element is not active, then at least one visual property of at        least the overlapping or potentially overlapping portion of the        first display element is adjusted.

Preferably, the means for adjusting adjusts at least one of colour,saturation, brightness or transparency of at feast the overlapping orpotentially overlapping portion of the first and/or second displayelement or the contrast between overlapping or potentially overlappingportions of the first and second display elements.

Preferably, the means for adjusting combines at least one visualproperty of the overlapping or potentially overlapping portions of thefirst and second display elements and the resultant combined visualproperty is applied to the overlapping portion of the active displayelement while at least one visual property of the other display elementis adjusted to minimise its effect on the image provided by the activedisplay element.

Alternatively, the means for adjusting adjusts at least one visualproperty of at least the overlapping portion of both the first andsecond display elements.

Preferably, the display controller also allocates the first and/orsecond display element to a selected display layer or layers based uponthe at least one determined visual property or upon an associatedpre-set display layer preference.

Preferably, the allocation to display layers is based upon storedpreferences associated either with a user of the multi-layer displaysystem or with particular software application programs run on acomputer system to which the multi-layer display system is connected.

Preferably, when a particular display layer includes a plurality ofdisplay elements, the display elements are arranged within that displaylayer to thereby control the way in which overlapping portions ofdisplay elements in a single display layer are displayed.

Preferably, when a plurality of display elements are dispersed across atleast two display layers, the display elements are arranged over the atleast two display layers thereby controlling the way in whichoverlapping portions of display elements on different display layers aredisplayed.

Preferably, the colours of at least the overlapping portion of the firstand second display elements are adjusted on a pixel-by-pixel basis.

In a third aspect, the invention consists in a method of usermanipulation of display elements in a multi-layer display systemincluding at least two overlapping display layers comprising the stepsof:

-   -   i) providing a user interface having a visual representation of        all of the display layers and symbols representing the display        elements on each display layer,    -   ii) allowing a user to select a symbol in the interface        representing a display element on a particular display layer,    -   iii) allowing the user to manipulate at least one property of        the selected symbol, and    -   iv) adjusting at least one property of the display element        represented by the selected and manipulated symbol in accordance        with the result of the manipulation of the at least one property        carried out on the selected symbol.

Preferably, the at least one property of the symbol manipulatable by auser includes the symbol's position in its present display layer, thedisplay layer in which the symbol is positioned, whether the displayelement is active and the order of the display element within itsdisplay layer.

Preferably, the step of allowing a user to select a symbol results inthe production of a display element selection identifier whichidentifies the particular display element to be manipulated.

Preferably, the movement of the selected symbol to a destinationrepresentation of a display layer in the user interface results in theproduction of a display layer selection identifier which identifies thedestination layer to which the display element, represented by theselected symbol, is to be transferred.

Preferably, the user interface is provided in a particular display layerand is a miniaturised two dimensional representation of the displaylayers and display elements.

Preferably, the user interface is provided in the same display layerthat a mouse pointer, movable by the user of the display system, isprovided.

Preferably, the visual representation of the at least two overlappeddisplay layers are shown side by side with no overlap.

Preferably, movement of a display element's symbol from the visualrepresentation of a first display layer to a visual representation of asecond display layer causes the display element represented by thesymbol to move from the first display layer to the second display layer.

Preferably, the position of a symbol within a visual representation of aparticular display layer corresponds to the position of the displayelement, represented by the symbol, within the particular display layer.

Preferably, the method of user manipulation also includes the displaycontrol method according to the first aspect.

In a fourth aspect, the invention consists in a multi-layer displaysystem comprising:

-   -   at least two overlapping display layers each of which are        adapted to depict display elements thereon,    -   a user interface having a visual representation of all of the        display layers and symbols representing the display elements on        each display layer,    -   means to allow a user to select a symbol in the interface        representing a display element on a particular display layer,    -   means to allow the user to manipulate at least one property of        the selected symbol, and    -   means for adjusting at least one property of the display element        represented by the selected and manipulated symbol in accordance        with the result of the manipulation of the at least one property        carried out on the selected symbol.

Preferably, the at feast one property of the symbol manipulatable by auser includes the symbol's position in its present display layer, thedisplay layer in which the symbol is positioned, whether the displayelement is active and the order of the display element within itsdisplay layer.

Preferably, the selection of a symbol in the user interface results inthe production of a display element selection identifier whichidentifies the particular display element to be manipulated.

Preferably, the movement of the selected symbol to a destinationrepresentation of a display layer in the user interface results in theproduction of a display layer selection identifier which identifies thedestination layer to which the display element, represented by theselected symbol, is to be transferred.

Preferably, the user interface is provided in a particular display layerand is a miniaturised two dimensional representation of the displaylayers and display elements.

Preferably, the user interface is provided in the same display layerthat a mouse pointer, movable by the user of the display system, isprovided.

Preferably, the visual representation of the at least two overlappeddisplay layers are shown side by side with no overlap.

Preferably, movement of a display element's symbol from the visualrepresentation of a first display layer to a visual representation of asecond display layer causes the display element represented by thesymbol to move from the first display layer to the second display layer.

Preferably, the position of a symbol within a visual representation of aparticular display layer corresponds to the position of the displayelement, represented by the symbol, within the particular display layer.

Preferably, a display controller in accordance with the second aspect isalso provided.

MLD units can be used to present various different types of displayelements on two or more display layers which are stacked or sandwichedor specially displaced from one another.

The present invention may facilitate a transfer of display elementsbetween the layers of a single multi-layer display. However it will beappreciated that the present invention may be used to transfer displayelements over a series of monitors or displays, both single layer and/ormulti-layer, driven from a single output device such as a computersystem. The device used to drive the MLD system may be a computersystem, but it should be appreciated that other types of output devicesand associated hardware may also be used in conjunction with the presentinvention.

Preferably the present invention may be adapted to provide a userinterface which allows a user of a multi-layer display system to adjustthe positioning or visual properties or organisation of display contentpresented by at least one multi-layer display. Such a user interfacesystem may allow a user of the display or displays involved to fullycustomise how display content of interest is displayed with respect tothe plurality of display layers available to them.

The present invention may be implemented through software code orcomputer based instructions loaded into a programmable logic device suchas a computer system or microprocessor. The method of operation or stepsexecuted by such software, in addition to the apparatus or systemprovided through physical hardware controlled by such software are alsowithin the ambit of the present invention.

Alternatively, the present invention may be completely electrically“hard-wired” in hardware wherein input display elements are allocated todisplay layers according to a pre-assigned allocation regime.Furthermore, the present invention may receive input informationrelating to the positioning or ordering of display elements on a displayadapter from hardware associated with the computer system. For example,the “Z-buffer” which is the section of memory in a video card whichkeeps track of which display elements can be viewed and which displayelements are hidden behind other display elements, may be accessed bythe present invention.

Preferably, a user interface implemented in conjunction with the presentinvention may have a number of different configurations depending on theparticular application it is to be used with. For example, the softwarecode or process implemented may form an operating system process, or canbe run as a background service on the computer system driving the MLDunit or units involved.

The present invention may be adapted to facilitate the assignment orallocation of a display point or points (that is, a “display element”)present within a wide variety of different types of display elements toa layer or layers of a multi-layer display system (that is, a displaysystem incorporating at least one MLD and optionally other MLD and/orSLD units). The invention may be used, for example, to assign a displaypoint or element on an SLD to a display layer on an MLD or vice-versa,or to assign a display point or element on a display layer of an initialMLD to a destination layer on that MLD or to a destination layer onanother MLD. Alternatively, the invention may be used to assign a newdisplay point or element which is yet to be displayed on any displaylayer to a display layer in the multi-layer display system.

The display elements can take many different forms, from for example,the various windows available under a graphical user interface forcomputer systems, to user input pointers such as mouse cursors, throughto in some instances, components of the user interface of a computeroperating system that displays information using the MLD involved.Furthermore, display elements present or embedded within Computer AidedDesign (CAD) systems or photographic or video media may also betransferred using the present invention.

A user driven and controlled transfer of GUI (Graphical UserInterface)-based windows, individual images, cursors, text pointersand/or other alternative forms of display elements in conjunction withthe present invention may provide significant advantages to users ofmulti-layer displays. This allows the content of display elements to beorganised and displayed across multiple layers for the effectivecommunication of the information content involved to the user orobserver of the display. Furthermore, the information involved may alsobe structured or organised to the benefit of a user across multiplelayers to assist in analytical investigations of data or information.

The at least one display element selection identifier may be generatedfrom or obtained by a user of the display interacting with or issuingcommands to the computer system which uses the current MLD to presentdisplay elements. These types of element selection actions executed byusers of the display can provide a trigger signal and also identityinformation with respect to a particular display element, such as awindow or mouse pointer for example, being selected for a transferbetween layers by the user.

The present invention may also incorporate one or more graphical userinterface display control components which allow such element selectionactions by a user to be captured, to in turn generate a display elementselection identifier. For example, in one preferred embodiment thepresent invention may encompass the provision of a control component inthe form of a toolbar type interface. This toolbar component may providea number of icons, buttons, pull-down menus and other similar interfacecomponents to allow a user to select and identify a particular displayelement or window for transfer to an alternate layer and/or to after itsvisual properties. A user's interaction with a control component canthen constitute an element selection action. In response to such elementselection actions, these types of control components may in turngenerate or supply the display element selection identifier to furthercomponents of the present invention which facilitate the actual transferof the display element involved to an alternate layer.

A single element selection action may in turn trigger the generation ofa single display element selection identifier only. A single action mayidentify a single element for transfer in such embodiments. However,those skilled in the art should appreciate that other implementations ofthe present invention where groups of related display elements orwindows may all be identified through a single element selection actionare also envisioned.

The present invention may also employ or use at least one display layerselection identifier which is generated by or sourced from a layerselection action completed by a user. In such embodiments a furthercontrol component may be displayed on the MLD involved. Such a commandcomponent can be employed by a user to select preferably a destinationlayer for a particular selected display element. Furthermore, in someinstances a single control component may also be used to generate bothdisplay element and display layer selection identifiers. In instanceswhere a single 2-layer MLD is employed in conjunction with the presentinvention, the use of display layer selection identifiers need not beconsidered to be essential. In such instances the selection of a displayelement of a window will result in the element being transferred to thealternate layer of the single display.

Once a display element selection identifier and associated display layerselection identifier have been received by the software employed inconjunction with the present invention, a transfer of the identifiedelement may be made to the identified display layer of the MLD involved.The control components employed as discussed above may allow a user toidentify a particular display element or window to transfer to aparticular display layer, and subsequently issue commands to execute thetransfer of the element identified to the layer identified.

Both identifiers for the display element and destination display layermay be supplied to graphics software and drivers employed in conjunctionwith the MLD system. These components may normally manage the display ofelements in the layers of the MLD system and on receipt of appropriatecommands issued in conjunction with the present invention, may executethe transfer between layers of the selected display element.

The present invention may provide many potential advantages over theprior art.

The present invention may be adapted to provide a user interfacefacility which will allow a user of the MLD unit and associated driverdevice (such as a computer system), to control the layers an whichspecific display elements are presented and/or the visual properties ofthose display elements.

The present invention may allow a user to nest, organise and in generalterms present specific components of display content depending on theirown preferences and preferably in the best possible configuration fordata analysis or information absorption purposes.

The present invention may also be implemented through software basedfacilities which can provide or employ recognisable command and controlbased components already familiar to users of computer systems whichemploy graphical user interfaces.

BRIEF DESCRIPTION OF DRAWINGS

Further aspects of the present invention will become apparent from thefollowing description which is given by way of example only and withreference to the accompanying drawings in which:

FIGS. 1a & 1 b is a block schematic diagram of the display layers of asingle multi-layer display device prior to and after the transfer of adisplay element between layers of the display using the presentinvention,

FIG. 2 is a flow diagram of information steps executed by softwareemployed in conjunction with a preferred embodiment of the presentinvention,

FIGS. 3A & 3B are schematic diagrams of a screen display in amulti-layer display system incorporating the user interface of thepresent invention, and

FIG. 4 is a block diagram of apparatus used in conjunction with the flowdiagram of FIG. 2.

BEST MODES FOR CARRYING OUT THE INVENTION Adjustment of VisualProperties and/or Location of Display Elements to Improve ViewingExperience

FIGS. 1a & 1 b illustrate a block schematic diagram of the displaylayers of a single multi-layer display device prior to and after thetransfer of a display element between display layers of the displayusing the present invention. Each display layer may comprise a LCDdisplay layer which is selectively transparent with the ability todisplay images thereon.

In the instance shown, a series of display layers 1 a, 1 b, 1 c arearranged and displaced in a line with respect to one another, however,in use, each of the layers are aligned, one directly in front of thenext to form a stacked or sandwiched construction. A single displayelement, shown in this embodiment as a graphical user interface window 2is presented on the front layer 1 a in the initial situation shown withrespect to FIG. 1a . Conversely, after a transfer made in conjunctionwith the present invention, the same display window has been transferredto the third rear display screen 1 c as shown in FIG. 1b . This window 2has either been repositioned after being selected by a user or observerof the display or automatically selected and moved, as has the displaylayer 1 c on which the window involved is to be displayed.

In a first preferred embodiment, the transfer of display elementsbetween layers and within layers is substantially automatically carriedout with no or minimal user input. In this way, the MLD seamlesslyappears to a user to act much like an SLD but with an improved visualexperience, for example, improved depth of field, improved contrast ofdifferent display elements due to the use of multiple display layerswhich are at different focal lengths from a user's eye.

This automatic control of the display of display elements on a displaysystem including at least one MLD firstly requires the detection orreception of display elements for display, for example, from the videooutput of a computer system. Visual properties, such as brightness,contrast and colour of each display element are determined and thedisplay elements are then allocated to appropriate display layers whichare known to produce the best viewing experience for a user. Forexample, it may be best to have bright images/windows presented behindor beneath dark images/windows. Thus, the control system may intuitivelyallocate display elements to selected display layers in such a way thatthe ability of the user to view the combined display elements ismaximised in a helpful and efficient manner.

Optionally, each user of the multi-layer display system could have astored set of viewing preferences in memory associated with a displaydevice or the computer system to which the multi-layer display isconnected. In this way, each user would consistently view multipledisplay elements in a preferred way. For example, a particular usercould prefer certain coloured display elements on a particular displaylayer or that user may prefer that brighter display elements arepresented in front of darker display elements, either within a singledisplay layer or over multiple display layers. These user preferencescould then be stored in a data file, updated and retrieved when requiredin order to usefully automatically allocate display elements to and/orwithin layers.

Similarly, preference data files relating to certain softwareapplications run on the computer system and viewed via the multi-layerdisplay device could be stored. In this way, the control system couldconsistently display software applications to all users. For example,MICROSOFT® WORD could have a preference file instructing the controlsystem to, for example, always display pop-up windows or dialogue boxeson the top layer of the multi-layer display device. The preference datafile for each software application could be user created/modified ormanufacturer created/modified.

Each display element in a graphical user interface in a multi-layerdisplay is conventionally associated with parameters of its “x, yposition”, “layer” (and/or “screen” in multi-screen systems) and“order”. The “order” parameter dictates which display element should bedisplayed in front of or behind any other display element when overlapof different display elements occurs. Overlap in the context of thisspecification means that, from a viewer's perspective, at least aportion of a first display element is obscured or blocked or occluded bya portion of a second display element (or by two or more other displayelements). However, because multi-layer display devices haveconventionally been treated as multiple stacked but separate devices, ithas been possible for a display element on a back layer which is active(that is, the display element which the user is currently interactingwith which is sometimes referred to as “in focus” or “in context”) tounintentionally be obscured by a non-active display element on a frontlayer.

The present invention treats the multiple layers of an MLD holisticallyrather than as separate devices and, because of the stacked orsandwiched interrelationship of the display layers between the user anda backlighting system, faces the above unique problems not encounteredin “mixed” display systems of multiple SLD units.

According to the present invention, the “order” parameter may bereplaced by an “order within layer” parameter and a further parameter,“viewing order” may be provided. The “order within layer” parameter isequivalent to the conventional “order” parameter but will only applywithin each respective layer. The new “viewing order” parameter willdetermine, over all display layers and over all interconnected displaydevices, which display element has precedence when overlap of displayelements occurs. In this way, the above-mentioned problem of blocking ofactive windows can be avoided. As mentioned above, the present inventionmay be utilised in various types of multi-layer display systemsincluding a single MLD, multiple interconnected MLD devices, orinterconnected single-layer display (SLD) devices and MLD devices.Preferably however, at least one MLD is incorporated into themulti-layer display system.

As mentioned above, a problem occurs when two display elements such asGUI windows, both displaying text, are positioned over each other (thatis, they overlap) in separate display layers of a multi-layer displaydevice. The result is that the text on both layers is very difficult (ifnot impossible) to read. “Text-On-Text” is the name which we have givento the problem however it is not limited to situations involving text.Overlaying text on a graphic image (or vice versa) can have the sameeffect of rendering the text difficult to read. More specifically, theproblem can be described as being at its worst:

-   -   when there is very little contrast between the overlapped        display elements or images displayed on different display        layers, and/or    -   when at least one display layer is displaying cluttered        information.

A solution which we have discovered to this problem is to use “alphablending” on a multi-layer display system to effectively effect thetransparency of, an a pixel by pixel basis, the information that is notactive (that is, not presently selected or “in context”), whether it ison a front or rear display layer. Transparency could, for example, beincreased by increasing the amount of white in the colour of a displayelement.

In computer graphics, each pixel is represented by three channels of8-bit colour information—red, green, and blue—and sometimes a fourthchannel called the alpha channel. The alpha channel controls the way inwhich other graphics information is displayed, such as the level oftransparency or opacity of the pixel. Alpha blending is the name forthis type of control, and it is used to simulate effects such as thevisual effect obtained by placing a piece of glass in front of an objectso that the object is either completely visible behind the glass,unviewable, or something in between. The alpha channel is really a maskwhich specifies how the pixel's colours should be merged with anotherpixel when the two are overlaid, one on top of the other.

The equation used in alpha blending in a single layer display is:[r,g,b] _(blended pixel) =α[r,g,b] _(foreground pixel)+(1−α)[r,g,b]_(background pixel)where [r, g, b] are the red, green, blue colour channels and a is aweighting factor.

The weighting factor is allowed to take any value from 0 to 1. When α isset to 0, the foreground pixel is completely transparent; when it is setto 1, the foreground pixel becomes opaque and totally obscures thebackground pixel; any intermediate value creates a mixture of the twopixel's “images”.

In a three layer multilayer display system for example, a single displaypoint on the front panel of the display unit actually has three separatepixels, arranged one in front of the other. Accordingly, conventionalalpha blending techniques which produce resultant RGB values for asingle pixel are inappropriate in a multi-layer display environmentwhich requires RGB values for each of the overlaid pixels in theseparate display layers. However, a similar principle can be invoked ina multi-layer display system in which the original colour values ofoverlaid pixels are combined in appropriate ratios to provide resultantRGB values for each resultant overlaid pixel. Alternatively, forexample, the term “alpha blending” as used herein may refer to themanipulation of a visual property (such as to increase transparency) ofpixels in non-active display elements in their respective display layerswhile the pixels in an overlapping active display element or elements intheir display layer or layers are not altered at all.

There are several ways that alpha blending can be used to overcome theText-on-Text problem in multi-layer display systems. For example:

-   -   1. the entire desktop (that is all display elements on all        display layers) could be faded or have their transparency        increased except for a particular display element (such as a        window) or elements which is/are in context, or    -   2. only a particular display element (such as a window or        windows) could have its transparency increased and everything        else remains unaffected, or    -   3. all objects on a particular display layer could have their        transparency increased.

In these ways, more light is thus able to be emitted from or through anout of context (or non-selected or non-active) display element so thatit has less effect on an overlapped in-context display element.

In any event, at least the overlapping portion of two overlapped displayelements may be alpha blended to a desired degree in order to ensurethat the user is best able to view the information from both layers.Because multi-layer display devices are designed to provide depthperception to a user, there individual display layers are spaced apart.As a result, depending upon the viewer's position in front of thedisplay device, the amount of overlap will appear to change. In order toaccount for this viewing angle/position variability, the presentinvention may alter a slightly larger portion of one or both of twooverlapped display elements.

It is also possible to allow a user to control the level of transparencyprovided by alpha blending the overlapped display elements.Alternatively, an optimum α value that minimises the Text-On-Text effectwhile still allowing the out of context text to be read may bepredetermined and defined as the preferred transparency point. The user(or software in an automated process) could then either set the text to“transparent” (which would initiate alpha blending with thepredetermined α value) or to opaque (no alpha blending) in order tosimplify operations for a user. A user could also specify that everytime a particular software application is opened its window is alphablended to a preset degree, such as transparent for example.

Research has also been carried out on a metric that, given a particularbackground and a particular font colour, can predict the readability ofthe text. That is, means for determining a level of interference asexperienced by a user between the first and second display elements. Inparticular, work has been carried out in developing a metric to predictthe readability or legibility of transparent or translucent text. In asituation where text is laid over some graphic background in amulti-layer display system, a software application could analyse, usingthe metric, whether the contrast between the two images was such thatthe text was difficult to read. If this analysis determined that it waslikely that the text was difficult to read then the program couldautomatically change the colour or transparency of the text (or thebackground, or both) to increase the contrast and make the text morereadable.

It is therefore possible for the control system of the present inventionto adjust the viewing properties (for example colour, brightness,transparency or contrast) of a particular display element in order toimprove the user's viewing experience. This could be necessary, forexample, if a display element is allocated for display on top of afurther display element, but because of its current visual properties(such as if the Text-on-Text problem existed), a user would find itdifficult or non-ideal to view the particular display element.

For example, as mentioned previously, a dark display element on a rear(further away from the viewing surface of the display device) displaylayer will reduce the amount of light available for illuminating anoverlapping display element on a front (closer to the viewing surface)display layer. In this case, even if the display element on the frontlayer is active, it will be at least partly occluded or obscured.Accordingly, the present invention would automatically adjust a visualproperty (such as the transparency or colour or brightness) of thedisplay element on the rear display layer, at least in the overlappingportion, so that the visibility of the active (front) display element isincreased.

In contrast, if a display element on a rear display layer is active andthere is a display element on a front display layer at least partiallyobscuring it, then a visual property (such as the transparency or colouror brightness) of the display element on the front layer may beautomatically adjusted, so that the visibility of the active (rear)display element is increased.

In both examples, a similar or improved result may be achieved byadjusting the visual properties of the display elements on both thefront and rear display layers. For example, if a display element in arear display layer was coloured red and a foreground, overlappingdisplay element was coloured green, then depending upon which displayelement was active, different proportions of the two colours could becombined and displayed on the front and/or back display layers, at leastin the overlapping portion of the display elements.

That is, if the foreground (red) display element was active, theoverlapping portion could be made to appear purple (that is thecombination of moistly red and a little green) by either not colouringthe overlapping portion at all in the rear display layer and colouringthe overlapping portion purple in the front display layer or byadjusting the colours of both display elements in the overlappingportion in such a way that the combination appears purple. However, itis necessary to bear in mind that the colours rendered in the twoseparate layers can not be considered totally independently in theircontribution to the resultant cumulative colour as detected by a viewerof the display—that is, the darker or less transparent the rear displayelement is, the less light that will be available to illuminate thefront display layer and therefore the front display element's appearanceis effected by the properties of the rear display element.

If however the background (green) display element was active, then theoverlapping portion could be made to appear a slightly reddish green(that is, a combination of mostly green and a little red). This could beachieved in either of the ways described above.

Alternatively or in conjunction with the above automatic adjustment ofvisual properties, the present invention also contemplates movingdisplay elements laterally, within their current display layer, in orderto remove overlap. Furthermore, a foreground display element in a frontdisplay layer may be moved wholly within a background display element ona rear display layer if the rear display layer has a suitably sized areaof appropriately rendered space. For example, if the background displaylayer had an area of white space within which the foreground displaylayer could fit, then the present invention may move the foregrounddisplay element over the white space area.

Although in the above description reference is made to overlap of twodisplay elements, it is of course possible that overlap occurs betweenmore than two display elements and the present invention aims to improvethe legibility or “viewability” of all overlapped portions. For example,more than one display element may be provided in a first layer, all ofwhich overlap with different portions and/or share common overlappingportions with a further display element on a different display layer.Alternatively, more than two display layers may be provided, each ofwhich includes a respective display element which overlaps with one ormore of the other display elements, either independently or sharing somecommon overlapping portion or portions.

Furthermore, in some case, display elements on two or more displaylayers may be rendering associated display elements which are linked toeffectively form a single display element. For example, an image may bedisplayed in “3-D” over multiple display layers so that movement of theany of the images display elements causes movement of all of theseparate associated display elements on the respective layers. In thiscase, if overlap occurs between the “3-D” image and a further displayelement, then any adjustment in position or visual properties requiredof the “3-D” image will be carried out an each of its constituentdisplay elements so that it is treated effectively as a single displayelement.

Interactive Visual Summary-User Interface

FIG. 2 illustrates a block schematic flowchart diagram of informationflows and steps executed by software employed in conjunction with afurther preferred embodiment of the present invention.

In initial step 3 the method executed consists of the software involveddetecting a display element selection action by a user and subsequentlygenerating at least one display element selection identifier inresponse. The detection of user selection actions can be completedthrough a receiving control component such as screen based toolbars, hotkey selections or radio buttons displayed on the MLD to the user on anyconvenient layer of the display.

One implementation of display element selection comprises the use of acontrol toolbar or interface which is in the form of a graphicalrepresentation which a user is able to interact with by, for example,dragging and dropping (using a mouse or touch screen for example) ontoareas representing the various display layers, icons or symbolsrepresenting various display elements on the multi-layer display system.For example, the graphical representation could be of a shelving system(such as book shelves) wherein each shelf represents a display layeronto which an icon representing a particular display element may bedropped to thereby instruct the allocation or assignment of that displayelement to the chosen display layer.

Another possibility would be to simply provide a miniaturised graphicalrepresentation of the various display layers and display elementsthereon in two dimensions. The various overlapped display layers wouldhowever be displayed side-by-side.

In a further alternative, the user interface could be provided as aminiaturised two dimensional representation of the appearance of thedisplay apparatus from the viewer's perspective. That is, the userinterface would include symbols representative of each of the displayelements shown on the various display layers although the user interfacewould be rendered in a single layer. In order to indicate to a user thedisplay layer to which each display element is currently assigned,different colours could be used for each display layer. For example, alldisplay elements on a front display layer could be coloured green whileall display elements on the next display layer could be coloured red forexample. Similarly, the outlines of symbols representing displayelements in different layers could be visually different (for example,solid line, dashed line, dot-dash line) or a small numeral (“1”, “2”,“3” for example) could be positioned within each symbol in the userinterface to identify its display layer.

Accordingly, the control toolbar component enables the user to select adisplay element and a display layer and to thereby generate a displayelement selection identifier and a display layer selection identifier.After a display element selection identifier has been generated for thedisplay element selected by a user, step 4 is then carried out in whichthe user manipulates the selected display element. The manipulation mayconsist in shifting the symbol representing a particular display elementto an alternative display layer. This layer selection action is detectedthrough the user interacting with a control component, which will inturn trigger the generation of a display layer selection identifierwhich identifies the selected destination display layer on which theuser wishes to have the selected display element presented. Aspreviously mentioned, in some MLD systems, (such as those containingonly two layers), this step may be optional. Alternatively, themanipulation may consist in making the selected display element active,altering its visual properties or changing its x, y position within itsoriginal display layer.

In the last step, step 5, the user manipulation carried out on thesymbol representative of the selected display element is transferred tothe actual display element so that it is moved to a selected displaylayer or it is made active or it's x, y location is changed for example.This is accomplished using graphics subsystems and software ordinarilyemployed by the MLD to display content on each of its layers based uponthe generated display element selection identifier and display layerselection identifier. In the case where the selected display element isto be transferred to a new destination display layer and this results inan overlap of display elements, the above described system ofautomatically adjusting the visual properties of the first and/or seconddisplay elements may be implemented.

Of course, the above described first (automatic allocation/alteration)and second (user controlled allocation) embodiments could be combined sothat some display elements within a multi-layer display system arealways automatically assigned while user input is used to assign otherdisplay elements.

FIG. 3A is a schematic diagram showing an example of how the userinterface according to the present invention may look in practice. Amulti-layer display system 6 includes for example two overlapped displaylayers, one directly in front of the other. A user interface 13 includesgraphical representations of first (or front) 14 and second (or back) 15display layers of the MLD system. Symbols 16A and 17A within the userinterface represent display elements 16B and 17B in the first displaylayer 14 while symbol 18A represents display element 18B in the seconddisplay layer 15. Display elements 16B and 17B may, for example, be GUIwindows. Display elements 16B, 17B and 18B may be easily andconveniently manipulated (moved within their layers, moved to differentlayers, made active/inactive or have their visual properties such ascolour, brightness, transparency or saturation) by a user interactingwith the representative symbols within user interface 13.

User interface 13 may be adjustable in size and may be moved to adifferent display layer. However, it may be advantageous if the userinterface were to automatically switch to the display layer in which theuser is currently working (that is, in which a selection element such asa mouse pointer is currently positioned). Alternatively, within the userinterface portion of the display apparatus, operation of a mouse pointerwithin any display layer could manipulate the symbols within the userinterface, even if the interface is positioned on a different layer thanthe layer in which the mouse pointer is currently positioned.

FIG. 3B is very similar to the schematic diagram of FIG. 3A but showsthe display system of FIG. 3A subsequent to a user selecting and moving(by dragging and dropping for example) symbol 16A to display layer 15.It can be seen that while within display layer 15 there is no overlap ofdisplay elements, display elements 16B and 17B are in fact overlappedand share a common overlapped portion 19. It will be appreciated thatthe present invention aims to improve the ease with which a viewer maydiscern information from or observe information within the overlappedportion 19 by either altering visual properties of the display elementsor adjusting their position(s).

FIG. 4 is a block diagram of a preferred embodiment of hardware forcarrying out the present invention. A display device 6 which maycomprise multiple interconnected multi- or single-layer devices isconnected to a control means 7 which controls the illumination of thevarious display layers of the display device. A user input device 8provides user input to the control means and a memory device 9 providesstorage for digital data files and executable software programs which,when executed, may control the operation of the control means 7.

Control means or controller 7 may comprise or include a microprocessorand also includes a position detector 10 for detecting or receiving theposition of a display element to be displayed on (all already beingdisplayed on) the display device and determining whether overlap willoccur between different display elements. The control means 7 alsoincludes an adjustment means 11 for automatically adjusting either theposition of a first and/or second overlapped display element or forautomatically adjusting at least one visual property of at least theoverlapping portion of the first and/or second display element. Theadjustment means may also determine whether the display element has anassociated pre-set display layer preference and may makes its adjustmentbased upon detected visual properties (such as colour, brightness orcontrast) of the overlapping display elements or upon a preset displaylayer preference or upon a determination that one or both layers containtext matter. A display driver 12 is also provided within the controller7 for generating appropriate electronic signals to the MLD system tocause the adjusted display elements to be rendered on their respectivedisplay layers of the display device 6.

It should be noted that the means 10, 11 and 12 could be provided byphysical hardware such as electronic circuits or could be provided bysoftware routines executed by the control means 7.

Aspects of the present invention have been described by way of exampleonly and it should be appreciated that modifications and additions maybe made thereto without departing from the scope thereof.

The invention claimed is:
 1. A method of displaying images in amulti-layer display system, the method comprising: providing amulti-layer display system including a first display screen and a seconddisplay screen, the first and second display screens overlapping eachother as viewed in a normal direction from a point of view of a viewer,and wherein the first display screen is configured to display a firstimage and the second display screen is configured to display a secondimage; displaying a first user interface on the second display screen,the first user interface comprises first and second adjacent sections,the first section configured to illustrate display elements displayedvia the first display screen and the second section configured toillustrate display elements displayed via the second display screen, andwherein the first user interface is configured to allow a user to move agiven display element from the first display screen to the seconddisplay screen by dragging and dropping that given display element fromthe first section to the second section of the first user interface;moving the first user interface from the second display screen to thefirst display screen, so that after said moving the first user interfaceis displayed on the first display screen and overlaps another displayelement that is displayed on the second display screen; after saidmoving, adjusting brightness and/or transparency of the first userinterface so as to improve visibility of the first user interface and/orthe another display element.
 2. The method of claim 1, wherein theanother display element comprises a second user interface.
 3. The methodof claim 2, wherein the first and second user interfaces each comprisetext.
 4. The method of claim 2, wherein the first and second userinterfaces each comprise a graphical user interface.
 5. The method ofclaim 1, further comprising moving the first user interface on the firstdisplay screen to a position where the first user interface does notoverlap the another display element.
 6. The method of claim 1, furthercomprising altering a size of the first user interface.
 7. The method ofclaim 1, further comprising alpha blending at least one of said firstuser interface and the another display element.
 8. The method of claim1, further comprising sharply focusing said first user interface whilede-focusing the another display element so as to improve visibility ofthe first user interface and/or the another display element.
 9. Themethod of claim 1, wherein the first and second display screens eachcomprise a liquid crystal layer.
 10. A multi-layer display systemcomprising: a first display screen and a second display screen, thefirst and second display screens configured to display a first image anda second image, respectively, which overlap each other as viewed in anormal direction from a point of view of a viewer; a controllerconfigured to: move a first user interface from the second displayscreen to the first display screen so that after the first userinterface is moved the first user interface is to be displayed on thefirst display screen and overlaps another display element that isdisplayed on the second display screen, wherein the first user interfacecomprises first and second adjacent sections, the first sectionconfigured to illustrate display elements displayed via the firstdisplay screen and the second section configured to illustrate displayelements displayed via the second display screen, and wherein the firstuser interface is configured to allow a user to move a given displayelement from the first display screen to the second display screen bydragging and dropping that given display element from the first sectionto the second section of the first user interface; adjust brightnessand/or transparency of the first user interface so as to improvevisibility of the first user interface and/or the another displayelement; and control symbols within the first user interface located onthe first display screen via manipulation of a pointer on the seconddisplay screen, so that the symbols of the first user interface can becontrolled by the pointer even when the pointer is on a differentdisplay screen.